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Interpretive  Guidance  Objectives 


To  understand  and  address 
the  issues  that  software 
organizations  have  when 
using  CMMI 

To  allow  current  SW-CMM 
users  to  more  easily  upgrade 
to  CMMI 

To  eliminate  as  many 
perceived  barriers  to  CMMI 
adoption  as  possible 
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Phase  I  Accomplishments 

Collected  comments  from  Birds-of-a-Feather  sessions  in 
conjunction  with  conferences  and  SPIN  meetings 

Formed  expert  group 

Received  responses  from  Web-based  questionnaire 

Received  limited  feedback  from  SCAMPI  appraisals 

Performed  preliminary  analysis  of  issues 

Released  Interpretive  Guidance  Preliminary  Report 
(available  at  http://www.sei.cmu.edu/cmmi/) 
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Phase  II 

The  purpose  of  Phase  II  is  to  analyze  issues  to  determine: 

•  if  interpretive  guidance  is  needed 

•  where  interpretive  guidance  is  appropriate 

•  what  form  interpretive  guidance  will  take 
At  a  minimum  we  will: 

•  perform  detailed  analysis  of  the  issues 

•  conduct  detailed  interviews  to  further  investigate  issues 

•  share  detailed  analysis  with  groups  at  the  SEI  to  understand 
how  their  activities  relate  to  identified  issues 

•  present  preliminary  data  at  conferences  and  SPIN  meetings  to 
validate  the  data  and  analysis 

•  produce  a  final  report  to  document  our  findings  and  conclusions 
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Detailed  Analysis 

Categorize  the  data. 

Identify  “low  hanging  fruit.” 

Identify  issues  that  will  be  addressed  by  the  Interpretive 
Guidance  project. 

Generate  change  requests  for  the  CMMI  Version  1.2 
revision  effort. 

Identify  issues  that  can  be  addressed  by  other  groups  at 
the  SEI. 
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Detailed  Interviews 

Follow-up  to  the  Interpretive  Guidance  Web-Based 
Questionnaire 

•  Clarify  and  elaborate  on  issues  identified  in  the 
questionnaire. 

•  Identify  potential  interpretive  guidance  artifacts  or  other 
solutions  for  the  community. 
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Detailed  Interview  Candidates 

Identified  21  organizations  as  candidates;  selected  the 
following  10  organizations: 

•  Automatic  Data  Processing 

•  Bank  of  America 

•  Electronic  Data  Systems 

•  Robert  Bosch 

•  Gartner  Group 

•  John  Hancock  Financial  Services 

•  Lockheed  Martin  M&DS 

•  Northrop  Grumman  IT 

•  McKesson  Corporation 

•  Raytheon  Space  and  Airborne 
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Detailed  Interview  Questions 

Tell  us  what  works  for  you  in  CMMI. 

Tell  us  what  does  NOT  work  for  you  in  CMMI. 

Let  us  know  about  obstacles  you  or  your  organization  have 
encountered. 

Show  us  how  you  and  the  organization  have/will  dealt/deal  with 
these  obstacles. 

Can  you  provide  examples  of  what  you  have  done? 

•  Templates,  Interpretation  Notes,  Policy  Guidelines 

•  Procedure  Notes,  Training  materials 
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Example  Issues  from  Detailed 
Interviews 

Project  Planning  and  Generic  Practice  2.2  --  Typical  work 
products  should  be  added,  as  it  is  convoluted  as  to  what  artifacts 
are  necessary.  We  had  a  proposal  that  showed  a  plan  to  do  plan 
for  the  program.  That  was  not  sufficient.  So  why  isn’t  a  proposal 
sufficient?  Eventually  it  was  accepted  after  explanation.  You 
need  a  typical  work  product  explicitly,  such  as  a  proposal 
development  process. 

Can  you  rewrite  the  MA  PA?  Rewrite  context  that 
captures/promotes  the  business  environment  so  we  understand 
what  the  objectives  are  that  customers  want  upfront.  We  don’t 
see  much  of  that  in  MA.  For  those  down  in  the  trenches,  what 
objectives  do  we  associate?  Objectives  of  the  business? 
program?  We  approached  it  as  objectives  of  the  business. 

Don’t  find  value-added  in  having  57  measures.  It’s  too  many. 
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Preliminary  Report 

Describes  the  data-collection  activities  from  both  BoF 
sessions  and  Web-based  questionnaire  efforts 

Includes  summaries  of  the  data  collected  through  August 
2003 
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Events  with  BoF  Sessions 

CMMI  Users  Group 
ICSPI  Conference 
New  York  City  SPIN 

QAAM/QAI  Conference  on  Managing  Software  Excellence 
PROFES  2002 

Acquisition  of  SW-Intensive  Systems 
SEPG  2003 

Southern  California  SPIN  meeting 
San  Diego  SPIN  meeting 
bITa  Europe  Conference 
NDIA  Transition  Workshop 
STC  2003 

European  SEPG  Conference 
Practical  Software  Measurement 
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Web-Based  Questionnaire 

Invited  participation  of  -7,000  people 

•  Over  4,000  people  had  direct  internet  access. 

•  Over  3,000  others  were  notified  that  the  questionnaire 
was  available. 

•  We  also  placed  an  announcement  on  the  SEI  Web 
site. 

The  number  of  individuals  responding  to  the  sections  of 
the  questionnaire  were 

•  Background  and  Context  (required  section)  -  668 

•  Global  Issues  -  587 

•  Generic  Goals  and  Generic  Practices  -  339 

•  Specific  Process  Areas  - 182 
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Background 

Nine  questions  were  asked  to  understand  the  background 
of  the  respondent. 

Some  questions  were  specific  to  the  person  filling  out  the 
questionnaire. 

Other  questions  were  providing  background  information 
about  the  organization. 
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How  would  you  best  describe  your  familiarity  with  CMMI? 


Didn't  respond 


Use  it  regularly 


Use  it  occasionally 


Heard  of  it 


Never  heard  of  it 

0%  10%  20%  30%  40%  50%  60%  70%  80% 

Percent  of  Respondents 
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What  if  any  CMMI  training  have  you  received? 

(Multiple  responses  were  permitted) 


SCAMPI  team  training 
SCAMPI  lead  appraiser  training 
CMMI  instructor  training 

Intermediate  CMMI 

Introduction  to  CMMI 


Percent  of  Respondents 
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Has  your  organization  made  a  decision  about 
adopting  CMMI? 

Didn't  respond 
Chosen  not  to  adopt  CMMI 
Well  institutionalized  in  organization 
Adoption  in  progress 
Decision  not  made  yet 

0%  10%  20%  30%  40%  50%  60%  70%  80% 

Percent  of  Respondents 
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Approximately  how  many  full-time  equivalent  (FTE)  employees 
does  your  organization  employ  who  are  primarily  engaged  in  the 
development,  maintenance,  or  acquisition  of  software  or 
software-intensive  systems? 


Percent  of  Respondents 
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How  would  you  best  describe  your  software  related  experience? 

In  what  application  domains  or  business  areas  have  your  worked? 

(Multiple  responses  were  permitted) 

Other 

Contractor  to  DOD/other  government 
DOD/other  government 
Commercial 
Custom  software 

Embedded,  real-time  systems 
Internet/Web/eCommerce 

IT,  IS,  MIS,  or  database 
Software  only 

0%  10%  20%  30%  40%  50%  60%  70%  80% 

Percent  of  Respondents 
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How  would  you  best  describe  your  familiarity 
with  the  Software  CMM? 

Didn't  respond 
Use  it  regularly 
Use  it  occasionally 

Heard  of  it 

Never  heard  of  it 

0%  10%  20%  30%  40%  50%  60%  70%  80% 

Percent  of  Respondents 
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Global 

Thirteen  questions  were  asked. 

General  questions  that  addressed  CMMI  adoption 
included 

•  CMMI  concepts  or  terminology 

•  model  representations 

•  costs 

•  ROI 
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In  your  opinion,  is  CMMI  adequate  for  guiding  process 

improvement? 


Didn't  Respond 
Don't  know 
Rarely  if  ever 
Sometimes 


More  often  than  not 
Almost  always 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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Adopting  CMMI  will  help  us  to  leverage  our  earlier  investments 

in  process  improvement. 


Didn't  Respond 
Don't  Know 
Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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Existing  CMMI  training  courses,  guidance  documents,  web 
resources,  and  other  process  assets  are  adequate  for  our  purposes. 


Didn't  Respond 
Don't  Know 
Strongly  Disagree 
Disagree 


Strongly  Agree 

0.0%  10.0%  20.0%  30.0%  40.0%  50.0% 

Percent  of  Respondents 
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Didn't  Respond 


Don't  Know 
Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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organization's  needs. 
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The  cost  of  adopting  CMMI  is  impeding  the  adoption  of  CMMI  in 

our  organization. 


Didn't  Respond 
Don't  Know 
Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 


©2003  by  Carnegie  Mellon  University 


page  26 


Carnegie  Mellon 

Software  Engineering  Institute 


Including  both  systems  engineering  and  software  in  a  single 

model  has  been  a  help  for  us. 


Didn't  Respond 


Don't  Know 

Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 


0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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We  have  had  difficulty  in  mapping  our  processes  to  the  CMMI. 

Didn't  Respond 

Don't  Know 

Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 


7% 

1 

'  |  15% 

18% 


Total  Respondents  =  587 


41% 


■ 


3% 
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We  have  had  difficulty  tracking  the  changes  and  additions  from 
models  that  we  have  previously  used. 


Didn't  Respond 


Don't  Know 

Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 

0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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Having  a  choice  between  the  two  model  representations  (staged 
or  continuous)  and  variations  (SW,  SE,  IPPD,  SS)  has  been  helpful  for  us. 


Didn't  Respond 
Don't  Know 


Strongly  Disagree 
Disagree 
Agree 
Strongly  Agree 


0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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Does  your  organization  need  ROI  or  other  quantitative  evidence 
to  help  make  the  business  case  for  adopting  CMMI? 


Didn't  Respond 


No,  it's  not  a  real  issue 
for  us 

No,  we've  already  built  a 
good  business  case 

Yes,  it  certainly  would 
help  to  have 


Yes,  we  must  have  it 


0%  10%  20%  30%  40%  50% 

Percent  of  Respondents 
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14% 
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Model  Components 

Seven  questions  were  asked. 

Questions  addressed 

•  confusing  words  or  phrases 

•  inappropriate  level  of  detail 

•  difficulty  of  application 

The  term  “comments”  is  used  to  show  where  a  respondent 
provided  information.  In  many  cases  this  information  did 
not  contain  an  issue. 

The  term  “issues”  is  used  where  there  is  a  comment  that  is 
either  positive  or  negative  and  can  be  analyzed. 
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Generic  Goals  (GGs)  and  Generic 
Practices  (GPs)  Issues 

There  were  979  comments  received;  -90%  contained 
issues. 

Many  issues  applied  to  the  product  suite  in  general,  not  just 
the  GGs  and  GPs. 

Some  examples  of  the  issues  included: 

•  During  SCAMPI  interviews,  how  specific  to  each  PA  must 
the  affirmations  for  GPs  be? 

•  GP  2.8  is  somewhat  redundant  with  M&A. 

•  GP  2.2:  What  comprises  a  minimum  acceptable  plan? 
Would  a  description  of  activities,  a  budget,  and  a  schedule 
be  considered  either  necessary  or  sufficient  (or  both)? 
These  are  not  explicitly  identified  as  either  necessary  or 
sufficient  under  GP  2.2  in  Chapter  4. 
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There  were  2,523  comments  collected;  783  were  issues 
(31%). 

OPD,  CAR,  OPF,  and  OID  received  the  fewest  issues. 

REQM,  PP,  and  SAM  received  the  most  issues.  However, 
these  were  the  first  three  PAs  that  respondents 
encountered  in  the  questionnaire. 

Issues  are  being  investigated  further  during  the  detailed 
interviews. 

Many  of  the  issues  have  been  submitted  as  change 
requests  for  CMMI  Version  1 .2. 

Other  issues  will  be  addressed  in  frequently  asked 
questions  (FAQs). 

For  a  few  issues,  interpretive  guidance  will  be  developed. 
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Examples  of  PA  Issues 

For  software  development,  the  practices  described  in  DAR 
will  not  have  to  be  applied  everyday!  The  relationships 
with  any  business  goal  are  not  obvious  for  software.  On 
the  other  side,  this  PA  describes  good  practices  for 
Systems  Engineering. 

For  Measurement  and  Analysis,  SP  2.3:  What  are 
"measurement  specifications,"  and  what  is  required  to 
manage  and  store  them? 

Breaking  out  REQM  and  RD  leads  to  confusion  for 
practicing  engineers.  Most  often  these  processes  for  the 
organization  are  defined  as  one.  This  makes  it  a  little  more 
difficult  to  evaluate  on  a  SCAMPI. 
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What  We’ve  Learned 

The  responses  were  overwhelmingly  positive. 

Many  of  the  issues  are  not  unique  to  commercial  software, 
IT,  and  IS  organizations. 

Many  of  the  issues  will  be  addressed  by  SEI  activities  that 
are  currently  underway: 

•  SCAMPI  B  and  C  development 

•  QA  activities 

•  Frequently  asked  questions  (FAQs) 

•  Technical  notes  and  articles 

•  VI. 2  revision 

•  Training  updates 
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What’s  Next 

Provide  additional  information  for  change  requests  already 
submitted  by  the  Interpretive  Guidance  project  for  the  VI  .2 
revision  effort. 

Generate  additional  change  requests  if  new  issues  are 
discovered. 

Identify  interpretation  issues  to  be  addressed  by  the 
creation  of  interpretive  guidance. 

Identify  positive  issues  that  can  be  shared  as  part  of  our 
“marketing”  communications. 
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A  final  Interpretive  Guidance  Report  will  be  published  in 
the  3rd  quarter  of  2004. 

Interpretive  guidance  information  will  be  developed  where 
necessary. 

Copies  of  the  preliminary  report  and  this  presentation  are 
available  on  the  CMMI  Website  at 

http://www.sei.cmu.edu.cmmi/adoption/interpretiveguidance.html 


Questions 
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For  More  Information  About  CMMI 

Go  to  CMMI  Web  site: 

http://www.sei.cmu.edu/cmmi 

http://seir.sei.cmu.edu 

Contact  SEI  Customer  Relations: 

Customer  Relations 
Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 
FAX:  (412)  268-5800 
customer-relations@sei.cmu.edu 
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Backup  Slides 


The  following  slides  provide  examples  of  the  issues  we 

collected  for  each  PA. 
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Causal  Analysis  and  Resolution  (CAR) 

51  comments  received;  9  of  these  were  issues 
Positive: 

•  Extending  the  scope  from  defects  to  other  problems 

•  Examples  and  typical  work  products  are  very  helpful 
Areas  for  Improvement: 

•  CAR  should  really  be  a  level  4  process  area  (PA).  Optimal 
causal  analysis  practices  are  required  at  level  4  (to  resolve 
causes  of  variation  from  expected/historical  performance) 
and  level  5  (to  fully  understand  the  gaps  between 
performance  baseline  and  performance  goals 

•  This  is  a  level  5  PA  and  therefore  must  be  driven  by  data.  I 
don't  believe  that  this  is  explained  well  within  the  model.  A 
better  overall  diagram  of  level-to-level  behavior  is  needed. 

•  This  PA  risks  having  people  think  that  root  cause  analysis 
does  not  apply  until  level  5. 

•  Typical  work  products  covering  other  problems  could  be 
improved. 
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Configuration  Management  (CM) 

135  comments  received;  33  of  these  were  issues 

Positive: 

•  Appropriate  content,  well  aligned  with  traditional  CM 
activities 

Areas  for  Improvement: 

•  Alignment  of  data  management  (DM)  versus  CM  is  needed 
due  to  handling  DM  in  Project  Planning  separately. 

•  Configuration  audits  are  frequently  confused  with  quality 
assurance  (QA)  audits,  especially  in  an  organization  that  still 
thinks  of  testing  as  a  QA  activity 

•  Baseline  audits  are  not  applicable  for  organizations  that  do, 
for  example,  only  studies  or  system  engineering  analyses 

•  Some  clarification  on  the  conceptual  boundary  between  this 
PA  and  REQM  would  be  helpful 
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Decision  Analysis  and  Resolution  (DAR) 

84  comments  received;  40  of  these  were  issues 

Positive: 

•  Structured  decisions  analysis  process  adds  immense  value  for 
organizational  level  decisions  such  as  new  technology. 

Initiatives,  growth  plans,  market,  new  tools  which  have  impact 
on  entire  organization 

Areas  for  Improvement: 

•  The  inclusion  of  DAR  as  a  process  area  gives  it  too  much 
emphasis.  It  seems  that  it  should  only  be  a  goal  in  another 
process  area,  or  somehow  be  considered  an  extension  to  the 
base  model 

•  For  software  development,  the  practices  described  will  not  have 
to  be  applied  everyday!  The  relationships  with  any  business  goal 
are  not  obvious  for  software.  On  the  other  side,  this  PA 
describes  good  practices  for  Systems  Engineering. 

•  Not  sure  how  to  unweave  TS,  DAR  and  RD  pieces  so  as  to  be 
able  to  tell  when  to  apply  which  one 
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Integrated  Project  Management  (IPM) 

62  comments  received;  23  of  these  were  issues 

Positive: 

•  Much  more  useful  that  ISM  in  CMM.  Has  a  lot  of  good  practices 
that  benefit  the  project  and  provide  ROI. 

•  Very  helpful  stakeholder  information. 

Areas  for  Improvement: 

•  There  is  confusion  that  has  arisen  in  many  appraisals  about  the 
relative  capabilities  indicated  by  the  two  goals.  There  is  no 
explicit  reference  to  a  "defined  process"  in  Goal  2,  so  it  is 
unclear  whether  the  collaboration/  cooperation  must  be  seen  in 
the  context  of  a  defined  process  or  simply  a  managed  process. 
As  a  result  it  is  common  to  have  ratings  of  "Not  Achieved"  for 
Goal  1  and  "Fully  Achieved"  for  Goal  2. 

•  "Integrated  plans"--unclearly  described. 

•  There  is  no  real  linkage  between  the  two  "normal"  goals  and  the 
IPPD  goals;  they  are  absolutely  separate.  There  is  no  reference 
to  a  "defined  process"  in  any  of  the  IPPD  material!  Some  effort 
needs  to  be  made  to  make  the  overall  content  in  the  IPPD 
extension  consistent. 
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Integrated  Supplier  Management  (ISM) 

60  comments  received;  18  of  these  were  issues 
Positive: 

•  Good  addition  to  SAM 
Areas  for  Improvement: 

•  PA  is  OK,  but  is  an  overkill  for  small  projects.  Do  the  activities 
defined  in  the  PA,  but  not  as  formal  as  required. 

•  Most  things  of  ISM  should  be  done  at  level  2. 

•  "Little  A"  acquisition  process  adds  very  little  value  over  SAM, 
and  does  not  address  the  process  content  needed  for  a  mature 
acquisition  organization  (as  in  SA-CMM).  There  is  insufficient 
value  of  this  PA  to  justify  its  adoption. 

•  Very  redundant  with  SAM  -  but,  at  least  it  was  easier  to  address 
that  way. 
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Integrated  Teaming  (IT) 

51  comments  received;  14  of  these  were  issues 

Positive: 

•  IT  PA  is  suitable  for  embedded,  real-time  systems. 

Areas  for  Improvement: 

•  This  is  one  PA  we  are  not  fond  of.  We  do  everything  in  the  PA, 
but  a  lot  more  informally.  This  PA  may  be  an  overkill  on 
teaming. 

•  Use  the  People  CMM  process  areas  as  needed  to  establish  the 
same  purpose. 

•  Team  charter  and  shared  vision  are  particularly  important  when 
the  team  members  are  coming  from  different  organizations.  But 
it  also  the  case  where  the  model  is  difficult  to  apply  and 
particularly  when  the  assessed  organization  is  only  a  component 
of  the  IPT  even  if  it  is  the  leader. 

•  Could  be  combine  with  PP  as  a  planning  PA. 
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Measurement  and  Analysis  (MA) 

167  comments  received;  67  of  these  were  issues 

Positive: 

•  Separating  M&A  into  a  separate  PA  is  one  of  the  most  powerful 
changes  from  the  CMMs,  since  it  highlights  the  integration  of 
business  objectives  and  goals  with  the  measurement  data 
collected,  analyzed,  and  reported.  Prior  implementations  of 
measurement  were  weak,  ineffective,  ambiguous,  and 
undirected. 

•  Actually  it  was  'Establish  Measurement  Objectives'  combined 
with  the  GP  'Plan  the  Process'  that  was  most  useful  as  we  had 
not  planned  this  process  sufficiently  before. 

Areas  for  Improvement: 

•  Useful  information  but  too  much  detail.  A  level  2  organization  is 
not  able  to  meet  this  criteria.  Too  costly  for  projects.  Not 
applicable  for  small  tasks  or  projects. 

•  SP  2.3:  What  are  "measurement  specifications",  and  what  is 
required  to  manage  and  store  them? 
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Organizational  Environment  for 
Integration  (OEI) 

59  comments  received;  23  of  these  were  issues 

Positive: 

•  Appreciate  the  inclusion  of  the  IPPD  concepts  into  the  model. 

Areas  for  Improvement: 

•  Too  wordy  and  has  a  lot  of  elements  that  we  feel  are  not 
necessary  or  should  not  be  required.  Management  in  particular, 
does  not  like  putting  the  incentive  for  integration  on  paper 

•  SP1 .2-1  -  need  some  more  specific  guidance  on  what  is  needed 
for  the  integrated  work  environment  and  what  alternatives  would 
satisfy. 

•  Combine  with  OT  under  a  Work  Environment  PA  to  reduce 
volume. 
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Organizational  Innovation  and 
Deployment  (OID) 

48  comments  received;  12  of  these  were  issues 

Positive: 

•  Glad  to  see  that  PCM  and  TCM  have  been  merged.  The  fact 
that  both  existed  in  the  CMM  made  little  sense. 

Areas  for  Improvement: 

•  TCM  was  diluted  by  the  way  it  has  been  implemented  in  CMMI. 

•  Concerns  on  the  de-emphasis  of  incorporation  of  new 
technologies  into  end  products.  This  will  be  a  missed 
opportunity  for  those  undertaking  process  improvement  in 
terms  of  the  benefits  and  results  they  will  report  on. 

•  The  Systems  Engineering  CMM's  Manage  Product  Line 
Evolution  provided  a  wonderful  perspective  on  the  need  to 
identify  and  evolve  the  products  provided  to  customers.  This  is 
missing  in  CMMI  and  references  to  product  in  OID  are  weak. 
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Organizational  Process  Definition  (OPD) 

52  comments  received;  4  of  these  were  issues 

Positive: 

•  Clear  definition  of  organizational  process  assets  has  been 
useful. 

Areas  for  Improvement: 

•  Never  seen  an  organization  achieve  level  2  without  a  Process 
asset  library.  That  portion  of  the  model  might  belong  in  level  2 

•  Combine  with  OPF  to  reduce  volume 

•  SP  1.3  in  many  cases  would  have  very  limited  applicability  with 
a  new  trend  that  is  emerging  -  'pre-tailored  lifecycles'  that  are 
proven  to  work 
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Organizational  Process  Focus 
(OPF) 

65  comments  received;  10  of  these  were  issues 
Positive: 

•  Well  aligned  with  OPF/OPD  from  SW-CMM  --  little  or  no 
transition  impact  for  organizations  that  already  have  process 
improvement  programs  in  place. 

Areas  for  Improvement: 

•  We  have  struggled  with  OPF  SP1 .1  and  MA  SP1 .1 .  These 
practices  need  to  be  integrated  and  supportive  of  each  other. 
However,  the  different  verbage  used  in  each  "process  needs" 
"information  needs"  do  not  always  map  easily. 

•  I  have  never  seen  an  organization  get  to  level  2  without  this. 
Not  sure  why  it  is  in  level  3. 

•  Combine  with  OPD  to  reduce  volume. 
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Organizational  Process 
Performance  (OPP) 

53  comments  received;  19  of  these  were  issues 

Positive: 

•  Merging  the  SW  CMM  material  for  SQM  and  QPM,  and  then 
splitting  them  based  on  what  the  organization  does  (OPP)  and 
what  the  project  does  (QPM)  was  a  very  effective  reorganization. 
It  has  made  implementation  of,  and  mapping  to,  the  material 
much  more  straightforward. 

Areas  for  Improvement: 

•  For  SP1 .2,  change  the  word  "Establish"  to  "Refine"  since  the 
process  measures  have  to  be  in  place  already  to  perform  this 
process  area.  It  is  not  a  matter  of  selecting  process  measures 
but  deciding  which  existing  measures  should  be  quantitatively 
managed. 

•  SP  1.4  and  SP  1.5  are  highly  confusing  ...  which  is  required  first, 
a  model  and  then  a  baseline  or  a  baseline  and  therefore  a 
model! 
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Organizational  Training  (OT) 

62  comments  received;  20  of  these  were  issues 

Positive: 

•  Like  the  separation  of  organizational  training  from  project 
training  (in  PP).  This  provides  greater  focus  within  the  PA,  and 
makes  it  easier  to  facilitate  adoption. 

•  SP  1.2  is  useful,  since  we  do  have  some  training  needs  that  are 
the  responsibility  of  the  organization,  and  some  that  are  the 
responsibility  of  the  projects 

Areas  for  Improvement: 

•  SP  2.3:  Are  class  evaluation  forms  filled  out  by  the  students 
sufficient  evidence  of  this  practice?  What  about  those  forms, 
plus  a  statistical  summary  of  the  data  on  these  forms?  What 
about  those  forms  and  the  summary,  plus  evidence  that  this 
summary  was  reviewed  by  those  responsible  for  the 
organizational  training  program? 

•  There  is  confusion  about  the  interpretations  of  the  relationship 
between  strategic  and  tactical  training  needs. 
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Product  Integration  (PI) 

83  comments  received;  26  of  these  were  issues 

Positive: 

•  Product  Integration  and  Build  was  a  neglected  area  in  CMM 

Areas  for  Improvement: 

•  Not  completely  clear  to  the  meaning  of  "sequence"  relative  to 
the  integration  of  product  or  product  components.  Example,  for 
“assemble",  it  is  described  as  the  assembly  of  the  products  or 
components.  In  software,  this  is  actually  accomplished  by  the 
use  of  scripts  to  automatically  perform  then  creation  of  the  load 
module  (or  "executable"  for  instantiation  during  product 
execution).  The  executable  is  then  verified  to  perform  its 
intended  purpose  according  to  requirement.  It  is  difficult  to  show 
this  "assembly"  process  results.  This  does  not  appear  to  be 
workable  for  large  scale,  software  intensive  projects. 

•  Too  many  references  to  product/  product  components  assembly 
vs.  software/  services. 

•  Considerable  redundancy  with  REQM,  DAR  and  CM. 
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Project  Monitoring  and  Control  (PMC) 

158  comments  received;  41  of  these  were  issues 
Positive: 

•  We  were  fortunate  to  have  most  of  the  PMC  covered  by  the 
preexisting  PMC  processes  developed  for  our  ISO  9001 
certification 

•  Helped  a  lot  to  better  focus  on  Quality 
Areas  for  Improvement: 

•  Could  clarify  what  is  intended  by  the  terms  "commitments", 
typical  implementations/artifacts,  and  how  they  are  established, 
monitored,  and  revised. 

•  Can  be  difficult  to  distinguish  between  risk  management  at  level 
2  (PP,  PMC)and  level  3  (RSKM).  In  my  opinion,  PMC  goes  too 
far  in  risk  mitigation  -  the  proactive  management  of  risks  is  best 
treated  at  ML3. 

•  It  seems  inconsistent  not  to  include  a  practice  for  tracking  the 
acquisition  of  needed  knowledge  and  skills  against  the  plan  for 
needed  knowledge  and  skills  developed  under  PP. 
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Project  Planning  (PP) 

197  comments  received;  91  of  these  were  issues 

Positive: 

•  The  move  to  attributes,  with  examples,  away  from  size 

•  Abandoning  critical  computer  resources  as  a  mandatory  element 

Areas  for  Improvement: 

•  Define  work  breakdown  structure  (WBS)  or  identify  what 
information  constitutes  a  WBS.  Define  what  goes  into  a  project 
plan.  Provide  more  examples  of  'attributes'  of  products.  Amplify 
information  about  Data  Management  Plan. 

•  Clearer  on  "size"  estimates;  are  they  required  (different  lead 
appraisers/consultant  interpret  the  model  differently) 

•  The  level  of  detail  available  for  explanation  of  SP1 .4  for  system 
engineering  projects  is  insufficient.  For  system  engineering 
projects,  engineering  judgments  may  also  be  a  good  method  of 
basis  of  estimates. 
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Process  and  Product  Quality 
Assurance  (PPQA) 

152  comments  received;  57  of  these  were  issues 

Positive: 

•  Adding  the  product  evaluations  to  this  PA.  Project  always 
confused  the  process  and  product  audits  so  now  they  are  doing 
both. 

Areas  for  Improvement: 

•  Our  quality  function  is  distributed  across  the  organization.  This 
fact  made  it  very  difficult  to  fulfill  this  process  area,  due  to  the 
interpretation  of  "objectivity".  There  was  difficulty  in  bringing  the 
assessment  team  to  agreement  that  a  distributed  quality  function 
could  be  objective. 

•  Redundant  with  verification  and  validation.  By  separating  these 
into  different  PAs,  you  have  added  cost  and  people  to  a  project. 
This  is  not  feasible  in  today’s  market. 
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Quantitative  Project  Management  (QPM) 

51  comments  received;  10  of  these  were  issues 

Positive: 

•  This  PA  allowed  us  to  focus  more  directly  on  process  and 
procedure  problems  and  improvements.  Quantitative  analysis 
quickly  separates  the  wheat  from  the  chaff 

Areas  for  Improvement: 

•  SP  1.2-1  has  been  somewhat  confusing.  Having  to  select 
specific  processes  based  on  process  capability  vs.  selecting 
processes  based  on  standards  that  have  worked  as  a  collective 
set  of  processes  has  led  to  a  number  of  discussions.  In  most 
cases,  the  latter  approach  is  probably  the  more  realistic 
approach. 

•  The  de-emphasis  of  using  control  charts  to  define  process 
performance  and  capability  was  a  mistake.  This  should  have 
been  clarified  and  emphasized. 

•  SP  2.2  &  SP  2.3  could  have  been  combined  since  they  are 
overlapping. 
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Requirements  Development  (RD) 

120  comments  received;  48  of  these  were  issues 

Positive: 

•  Introduced  in  our  organization  better  defined  or  new  concepts 
(e.g.,  operational  scenarios,  non-functional  requirements, 
elicitation,  validation) 

•  Gives  a  good  road  map  on  capturing,  analyzing  and  establishing 
requirements 

Areas  for  Improvement: 

•  Why  are  there  validation  steps  part  of  the  process  areas  and  yet 
there  is  still  a  validation  PA?  How  do  they  map? 

•  SP  3.4-3  achieve  balance  -  when  do  you  determine  that  balance 
achieved? 

•  SP  1 .4  &  SP  1 .5  could  have  been  combined  as  1 .5  is  a  logical 
step  which  could  be  done  in  1.4  itself. 
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Requirements  Management  (REQM) 

249  comments  received;  91  of  these  were  issues 

Positive: 

•  SYSTEMS  +  SOFTWARE  =  GREAT 

•  Traceability  has  finally  been  directly  addressed 

Areas  for  Improvement: 

•  Some  strong  redundancies  with  configuration  management  here. 
REQM  looks  like  some  kind  of  "specialization"  of  CM.  It  is  not  so 
easy  to  work  with  these  redundancies 

•  Bi-directional  traceability  could  be  better  explained,  with 
examples 

•  Horizontal  versus  vertical  traceability  can  be  explained  better 

•  Breaking  out  REQM  and  RD  leads  to  confusion  for  practicing 
engineers.  Most  often  these  processes  for  the  organization  are 
defined  as  one.  This  makes  it  a  little  more  difficult  to  evaluate  on 
a  SCAMPI. 
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Risk  Management  (RM) 

87  comments  received;  27  of  these  were  issues 
Positive: 

•  Very  good  addition  to  model.  Focus  on  Risk  Management  as  a 
stand  alone  process  area  gives  needed  focus. 

•  This  PA  will  be  one  of  the  most  useful  PAs  in  the  model. 

Areas  for  Improvement: 

•  Clarify  difference  in  RSKM  with  respect  to  risk  identification  and 
tracking  in  PP  and  PMC. 

-  Although  the  specific  practices  in  RSKM  should  be  done 
according  to  the  Risk  Taxonomy  established  in  SGI ,  it  is  still 
redundant  as  at  a  CL1  for  RSKM,  this  could  be  the  same  as 
SP2.2  in  PP. 

•  Could  be  combined  with  DAR  under  a  decision  making  process 
area. 
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Supplier  Agreement  Management  (SAM) 

197  comments  received;  91  of  these  were  issues 

Positive: 

•  Obviously  a  vast  step  above  the  SCM  of  SW  CMM 

•  Goals  and  practices  are  well  aligned  with  typical  industry 
processes  for  supplier  selection  and  monitoring. 

Areas  for  Improvement: 

•  Both  SAM  and  ISM  neglect  an  important  topic:  procurement 
planning 

•  Is  purchasing  from  a  catalog  a  supplier  agreement? 

•  The  sudden  inclusion  of  COTS  in  SG2  seems  a  little  out  of  place. 
Need  to  clarify  the  concepts  of  how  COTS  applies  and  fits  into 
this  PA  (and  relationship  with  other  PAs,  TS  etc.) 

•  SP2.1-1  should  be  in  goal  1 
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Carnegie  Mellon 

Software  Engineering  Institute 

Technical  Solution  (TS) 

101  comments  received;  39  of  these  were  issues 

Positive: 

•  Improves  the  way  project  managers  and  engineers  judge  their 
technical  solutions.  Gets  away  from  running  a  one  man  show 
with  only  that  person’s  ideas. 

Areas  for  Improvement: 

•  Not  sure  how  to  unweave  TS,  DAR  and  RD  pieces  so  as  to  be 
able  to  tell  when  to  apply  which  one 

•  Very  difficult  to  map  to  a  service  environment.  Most  of  the  work 
is  of  2  -  5  days  duration.  You  will  not  be  evaluating  alternatives. 

•  SP1.2  -  practice  is  redundant  -  in  at  least  one  industry  guideline, 
"operational  concept"  includes  scenarios,  environments, 
conditions,  operating  modes,  operating  states,  and  much  more. 
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Carnegie  Mellon 

Software  Engineering  Institute 

Validation  (VAL) 

89  comments  received;  45  of  these  were  issues 

Positive: 

•  The  introduction  of  this  PA  is  extremely  useful  to  explain  to 
people  what  it  is  all  about  and  the  added  value  on  top  of 
verification. 

•  Definition  of  validation  (purpose  and  introductory  notes) 

Areas  for  Improvement: 

•  Separating  Validation  from  Verification  was  a  mistake.  In 
practice,  many  organizations  are  not  specifically  responsible  for 
Validation. 

•  SP  1 .1 :  Can  validation  be  applicable  to  interim  work  products  as 
well  as  the  final  "products  and  product  components"?  This  is 
mentioned  in  the  Validation  PA  Introductory  Notes,  but  not  here. 
Suggest  that  if  applicable,  it  should  be  explicitly  mentioned  in 
this  practice  and/or  the  elaboration  of  this  practice. 
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Carnegie  Mellon 

Software  Engineering  Institute 

Verification  (VER) 

90  comments  received;  43  of  these  were  issues 
Positive: 

•  Very  useful  PA  for  project  with  safety  constraints 
Areas  for  Improvement: 

•  Lot  of  literature  talks  about  Verification  and  Validation  together. 
Also,  some  organization  perform  V&V.  In  such  situations,  how 
can  these  PAs  be  interpreted  separately  and  implemented? 

•  Sometimes  difficult  to  separate  the  evidence  for  PI  vs.  VER  vs. 
VAL  because  they  are  often  done  in  the  same  tests. 

•  Need  to  define  inspections,  structured  walkthroughs  and  active 
reviews  in  Glossary 

•  Its  confusing  from  the  standpoint  that  peer  reviews  are  a  form  of 
verification  (a  way  to  verify)  and  they  are  called  out  separately 
even  when  they  should  be  subsumed  under  the  other  goal 
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